Multi-Cluster Container Replication

ABSTRACT

An improved scalable object storage system allows multiple clusters to work together. In one embodiment, a multi-cluster synchronization system synchronizes data between two or more clusters. Each cluster has a cluster-internal network, with object storage services and container services. The container services track and replicate metadata associated with the object storage service. An intercluster network connects the two clusters and performs a one-way synchronization of the objects and metadata associated with a particular container. In a further embodiment, multiple synchronization relationships can be set up, either in a cycle (with two or more participants), in a line, or in a tree. For example, the multi-cluster replication could be used to transparently synchronize objects in a CDN network.

This is a continuation application of and claims priority to the U.S.patent application Ser. No. 13/089,442 filed Apr. 19, 2011, entitled“Massively Scalable Object Storage System,” and claims benefit of U.S.provisional patent application 61/450,166, filed Mar. 8, 2011, entitled“Massively Scalable File Storage System.” This application is alsorelated to co-pending non-provisional U.S. patent application Ser. Nos.13/089,476, 13/089,487, and 13/089,510, all filed Apr. 19, 2011. Theentirety of these disclosures is hereby incorporated herein byreference.

BACKGROUND

The present disclosure relates generally to cloud computing, and moreparticularly to a massively scalable object storage system to providestorage for a cloud computing environment. Cloud computing services canprovide computational capacity, data access, networking/routing andstorage services via a large pool of shared resources operated by acloud computing provider. Because the computing resources are deliveredover a network, cloud computing is location-independent computing, withall resources being provided to end-users on demand with control of thephysical resources separated from control of the computing resources.

As a term, “cloud computing” describes a consumption and delivery modelfor IT services based on the Internet, and it typically involvesover-the-Internet provisioning of dynamically scalable and oftenvirtualized resources. This frequently takes the form of web-based toolsor applications that users can access and use through a web browser asif it were a program installed locally on their own computer. Detailsare abstracted from consumers, who no longer have need for expertise in,or control over, the technology infrastructure “in the cloud” thatsupports them. Most cloud computing infrastructures consist of servicesdelivered through common centers and built on servers. Clouds oftenappear as single points of access for consumers' computing needs, and donot require end-user knowledge of the physical location andconfiguration of the system that delivers the services.

Because the flow of services provided by the cloud is not directly underthe control of the cloud computing provider, cloud computing requiresthe rapid and dynamic creation and destruction of computational units,frequently realized as virtualized resources. Maintaining the reliableflow and delivery of dynamically changing computational resources on topof a pool of limited and less-reliable physical servers provides uniquechallenges. Accordingly, it is desirable to provide a better-functioningcloud computing system with superior operational capabilities.

SUMMARY

In one embodiment, a trust and federation relationship is establishedbetween a first cluster and a second cluster. This is done bydesignating a first cluster as a trust root. The trust root receivescontact from another cluster, and the two clusters exchangecryptographic credentials. The two clusters mutually authenticate eachother based upon the credentials, and optionally relative to a thirdidentity, authorization, or authentication service. Following theauthentication of the two clusters, a service connection is establishedbetween the two clusters and services from the remote cluster areregistered as being available to the cluster designated as the trustroot. In further embodiments, a multi-cluster gateway is designated asthe trust root, and the two clusters can be mutually untrusting. In athird embodiment, the remote cluster can be also designated as a trustroot, and two one-way trust and federation relationships can be set upto form a trusted bidirectional channel.

When a trusted connection is set up between the two clusters, a userworking with the first cluster, or with a multi-cluster gateway, can askfor services and have the request or data transparently proxied to thesecond cluster. Cross-cluster replication is one anticipated service, asare multi-cluster compute or storage farms based upon spot availabilityor various provisioning policies. For example, a vendor providing acloud storage “frontend” could provide multiple backends simultaneouslyusing the trust and federation relationship.

In one embodiment, a multi-cluster gateway can have a two, three, orhigher-level ring that transparently matches an incoming request withthe correct cluster. In the ring, a request is first mapped to anabstract “partition” based on a consistent hash function, and then oneor more constrained mappings map the partition number to an actualresource. In another embodiment, the multi-cluster gateway is a dumbgateway, and the rings are located only at the cluster level.

Various embodiments use existing cryptographic or authenticationprotocols when exchanging tokens or verifying each other; sharedsecrets, a public/private keypairs, a digital certificates, Kerberos,XAUTH and OAUTH are all contemplated. Separate authentication entitiesare also contemplated, such as an OpenID provider, LDAP store, or RADIUSserver.

In another embodiment, there is a multi-cluster synchronization systembetween two or more clusters. Each cluster has a cluster-internalnetwork, with object storage services and container services. Thecontainer services track and replicate metadata associated with theobject storage service. An intercluster network connects the twoclusters and performs a one-way synchronization of the objects andmetadata associated with a particular container. This can be done eitherthrough the direct association of the container and object storageservices, such as through a trust and federation relationship, or it canbe opaque, so that the cross-cluster replication treats the remoterepository as a black box and uses the external API to call andmanipulate the files.

In a further embodiment, multiple synchronization relationships can beset up, either in a cycle (with two or more participants), in a line, orin a tree. For example, the multi-cluster replication could be used totransparently synchronize objects in a CDN network.

In another embodiment, the multi-cluster synchronization system usesvariable compression to optimize the transfer of information betweenmultiple clusters. Aside from the simple use of compression to minimizethe total number of bytes sent between the two clusters, the size of theobjects sent across the wire can be dynamically changed using filecompression to optimize for higher throughput after considering packetloss, TCP windows, and block sizes. This includes both the packaging ofmultiple small files together into one larger compressed file, saving onTCP and header overhead, but also the chunking of large files intomultiple smaller files that are less likely to have difficulties due tointermittent network congestion or errors. Depending on the state of thenetwork and disks, the best size can vary; examples range fromapproximately 4 MB (largest non-fragmented packet using jumbo frames) to64 MB (block size on some distributed filesystems) to 1 GB and above. Afurther embodiment uses forward error correction to maximize the chancesthat the remote end will be able to correctly reconstitute thetransmission.

According to another embodiment, the improved scalable object storagesystem includes a distributed information synchronization system,comprising a first subsidiary node coupled to a network, the firstsubsidiary node including a first non-transitory computer-readablemedium wherein the first computer-readable medium includes a firststructured information repository, and wherein information in the firststructured information repository is subject to internal consistencyconstraints; a second subsidiary node coupled to a network, the secondsubsidiary node including a second non-transitory computer-readablemedium wherein the second computer-readable medium includes a secondstructured information repository, and wherein information in the secondstructured information repository is subject to internal consistencyconstraints; a repository synchronizer coupled to the first and secondstructured information repositories; the repository synchronizer furtherincluding a consistency evaluation module adapted to evaluate thedifferences between the first structured information repository and thesecond structured information repository; an internal modificationmodule adapted to modify the internal structures of a structuredinformation repository; an external replication module adapted to deletea target structured information repository and replace it with areplicated copy of a source structured information repository; and athreshold comparator; wherein the repository synchronizer is adapted toevaluate the first and second structured information repositories anddetermine a level of difference and compare the level of difference to aconfigurable threshold using the threshold comparator; if the level ofdifference is above the configurable threshold, modify the internalstructures of a selected structured information repository using theinternal modification module; and if the level of difference is belowthe configurable threshold, delete the selected structured informationrepository and replace it with a replicated copy of a consistentstructured information repository using the external replication module.

According to another embodiment, the improved scalable object storagesystem includes a method for synchronizing structured information in adistributed system, comprising storing a first structured informationrepository on a first non-transitory computer-readable medium, whereininformation in the first structured information repository is subject tointernal consistency constraints; storing a second structuredinformation repository on a second non-transitory computer-readablemedium, wherein information in the second structured informationrepository is subject to internal consistency constraints; evaluatingthe differences between the first structured information repository andthe second structured information repository to determine a preferredstate and a difference measurement quantifying a difference from thepreferred state; determining whether the difference measurement exceedsa configurable threshold; modifying a selected structured informationrepository if the difference measurement for the selected structuredinformation repository is less than the configurable threshold, whereinthe modification of the selected structured information repository issubject to the internal consistency constraints of the selectedstructured information repository, deleting the selected structuredinformation repository if the difference measurement for the selectedstructured information repository is greater than the configurablethreshold, and replacing the selected structured information repositorywith a replica of a structured information repository in the preferredstate, wherein either modifying the selected structured informationrepository or deleting and replacing the structured informationrepository changes the non-transitory computer-readable medium storingthe selected structured information repository such that the selectedstructured information repository is both compliant with its internalconsistency constraints and in the preferred state. The method may alsoinclude determining that both the first structured informationrepository and the second structured information repository are not inthe preferred state; pre-selecting the structured information repositorythat is closer to the preferred state and modifying the pre-selectedstructured information repository to bring the pre-selected structuredinformation repository to the preferred state, subject to the internalconsistency requirements of the pre-selected structured informationrepository, regardless of the configurable threshold.

According to another embodiment, the improved scalable object storagesystem includes a non-transient computer readable medium containingexecutable instructions, which when executed on a processor update afirst structured information repository on a first non-transitorycomputer-readable medium, subject to internal consistency constraints;update a second structured information repository on a secondnon-transitory computer-readable medium, subject to internal consistencyconstraints; evaluate the differences between the first structuredinformation repository and the second structured information repositoryto determine a preferred state and a difference measurement quantifyinga difference from the preferred state; determine whether the differencemeasurement exceeds a configurable threshold; modify a selectedstructured information repository if the difference measurement for theselected structured information repository is less than the configurablethreshold, subject to the internal consistency constraints of theselected structured information repository, delete the selectedstructured information repository if the difference measurement for theselected structured information repository is greater than theconfigurable threshold, and replace the selected structured informationrepository with a replica of a structured information repository in thepreferred state.

According to another embodiment, the improved scalable object storagesystem includes a non-transient computer readable medium containingexecutable instructions, which when executed on a processor update afirst structured information repository on a first non-transitorycomputer-readable medium, subject to internal consistency constraints;update a second structured information repository on a secondnon-transitory computer-readable medium, subject to internal consistencyconstraints; evaluate the differences between the first structuredinformation repository and the second structured information repositoryto determine a preferred state and a difference measurement quantifyinga difference from the preferred state; determine whether the differencemeasurement exceeds a configurable threshold; modify a selectedstructured information repository if the difference measurement for theselected structured information repository is less than the configurablethreshold, subject to the internal consistency constraints of theselected structured information repository, delete the selectedstructured information repository if the difference measurement for theselected structured information repository is greater than theconfigurable threshold, and replace the selected structured informationrepository with a replica of a structured information repository in thepreferred state.

The specifics of these embodiments as well as other embodiments aredescribed with particularity below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view illustrating an embodiment of a file storagesystem.

FIG. 122 is a schematic view illustrating an embodiment of aninformation handling system used in the file storage system of FIG. 1.

FIG. 3 is a schematic view illustrating an embodiment of a logicalstructure provided by the file storage system of FIG. 1.

FIG. 4 a schematic view illustrating an embodiment of a logicalstructure provided by the multi-cluster storage system encompassingmultiple similar file storage systems.

FIG. 5 is a schematic view illustrating an embodiment of a user account.

FIG. 6 is a flow chart illustrating an embodiment of a method forstoring an object.

FIG. 7 is a flow chart illustrating an embodiment of a method forcreating a ring.

FIG. 8 is a flow chart illustrating an embodiment of a method forreassigning partitions in a ring.

FIG. 9 is a flow chart illustrating an embodiment of a method forassigning replication and trust targets between containers.

FIG. 10 is a schematic view illustrating a first embodiment of acontainer replication structure.

FIG. 11 is a schematic view illustrating a second embodiment of acontainer replication structure.

DETAILED DESCRIPTION

The following disclosure has reference to an object and file storageservice delivered on top of a cloud architecture.

Referring now to FIG. 1, an embodiment of a file storage system 100 isillustrated. The file storage system 100 includes a user device 102connected to a network 104 such as, for example, a Transport ControlProtocol/Internet Protocol (TCP/IP) network (e.g., the Internet.) Astorage management server 106 is connected to the network 104 and to aplurality of storage servers 108. While only one user device has beenillustrated as connected to the network 104 for clarity of discussion,one of skill in the art will recognize that a plurality of user devicesmay, and typically will, be connected to the network 104. While only onestorage management server coupled to a plurality of storage servers hasbeen illustrated as connected to the network 104 for clarity ofdiscussion, one of skill in the art will recognize that a plurality ofstorage management servers, each connected to a plurality of storageservers may, and typically will, be connected to the network 104. Eachof the user device 102 and the storage management server 106 includes arespective network interface for communicating with the network 104(e.g., outputting information to, and receiving information from, thenetwork 104).

Each of the user device 102, storage management server 106, and theplurality of storage servers 108 may include a respective informationprocessing system, a subsystem, or a part of a subsystem for executingprocesses and performing operations (e.g., processing or communicatinginformation). An information processing system is an electronic devicecapable of processing, executing or otherwise handling information, suchas a computer.

Referring now to FIG. 2, an information processing system 110 which isrepresentative of one of, or a portion of, the information processingsystems described above, is illustrated. The information processingsystem 110 may include any or all of the following: (a) a processor 112for executing and otherwise processing instructions, (b) acomputer-readable medium 120, which is operably coupled to the processor114, for storing information, as discussed further below, and (f)various other electronic circuitry for performing other operations ofthe information processing system 110 known in the art. For example, theinformation processing system 110 may include a network interface (e.g.,circuitry) for communicating between the processor 110 and the network104 and/or other devices, and (b) a memory device (e.g., FLASH memory, arandom access memory (RAM) device or a read-only memory (ROM) device forstoring information (e.g., instructions executed by processor 112 anddata operated upon by processor 112 in response to such instructions)).In some embodiments, the information processing system may also includesystems suitable for in-person use, such as a one or more input devices114, a display device 116, or a print device 118. The use of an inputdevice 114, display device 116, or print device 118 is not necessary andshould not be construed as limiting.

The computer-readable medium 120 and the processor 110 are structurallyand functionally interrelated with one another as described below infurther detail, and information processing system of the illustrativeembodiment is structurally and functionally interrelated with arespective computer-readable medium similar to the manner in which theprocessor 110 is structurally and functionally interrelated with thecomputer-readable medium 120. As discussed above, the computer-readablemedium 120 may include a hard disk drive, a memory device, and/or avariety of other computer-readable media known in the art, and whenincluding functional descriptive material, data structures are createdthat define structural and functional interrelationships between suchdata structures and the computer-readable medium 120 (and other aspectsof the system 100). Such interrelationships permit the data structures'functionality to be realized. For example, the processor 112 reads(e.g., accesses or copies) such functional descriptive material from thecomputer-readable medium 120 onto the memory device of the informationprocessing system 110, and the information processing system 110 (moreparticularly, the processor 112) performs its operations, as describedelsewhere herein, in response to such material stored in the memorydevice of the information processing system 110. In addition to readingsuch functional descriptive material from the computer-readable medium120, the processor 112 is capable of reading such functional descriptivematerial from (or through) the network 104. In one embodiment, thecomputer-readable medium is non-transitory.

Referring now to FIG. 1 and FIG. 3, one embodiment of the file storagesystem of FIGS. 1 and 2 has the logical structure 200 as shown in FIG.3. The logical structure 200 includes a user 202 connected to a proxy204. In one embodiment, the user 202 may be provided by the user device102, the proxy 204 may be provided by the storage management server 106,and the user 202/proxy 204 connection may be created by the coupling ofthe user device 102 to the storage management server 106 through thenetwork 104. The proxy 204 is connected to one or more rings 206 such asan object ring 206 a, a container ring 206 b, and an account ring 206 c,described in further detail below, that are connected to an objectservice 208, container service 210, and an account service 212,respectively, described in further detail below. In other embodiments,there are other types of objects managed by rings, such as a structureddata ring, a graph storage ring, or another type of ring (not pictured).In such embodiments, each ring would be connected to an appropriateservice, such as a structured data service, a graph service, or anotherservice (not pictured).

Each of object service 208, the container service 210, and the accountservice 212 are connected to a plurality of storage pools 214. In oneembodiment, the rings 206 may include software that is stored on acomputer-readable medium location in the storage management server 106and/or the storage servers 108. In one embodiment, the object service208, the container service 210, and the account service 212 may includesoftware that is stored on a computer-readable medium located in thestorage management server 106 and/or the storage servers 108. In oneembodiment, the storage pools 208 may be provided by the storage servers108. In one embodiment, the proxy 204/rings 206/object service208/container service 210/account service 212/storage pool 214connections may be created by the connection of the storage managementserver 106 with the storage servers 108. In a further embodiment, therings are implemented at least in part using electrical circuits on asemiconductor chip to achieve better speed and latency.

In one embodiment, each storage pool 214 is provided by a separatestorage server 108 or includes a virtual server that is included in aportion of one of the storage servers 108 or across a plurality of thestorage servers 108. For example, the storage servers 108 may bephysically located in one or more data centers, and the resources of thestorage servers 108 may be virtualized according to the requirements ofa plurality of users (e.g., the user 202) such that the plurality ofstorage pools 214 are provided to the plurality of users in order tostore files and/or data objects. Thus, resources for a particularvirtual server or storage pool may span across multiple storage servers108.

Referring now to FIG. 4, a multi-cluster file storage system is shown atreference 240. The multi-cluster file storage system 240 encompassesmultiple clusters 250 a-250 n. These clusters may be included inseparate regions, such as the exemplary regions 252 a and 252 b. Eachcluster may also be under the control of a separate organization. Eachcluster includes a file storage system, such as the file storage systemdescribed relative to FIG. 3, possibly including in each file storagesystem a proxy 254 a-n, one or more rings 256 a-n, object, container,account, or other services 258 a-n, and a storage pool 260 a-n. In oneembodiment, the user 202 interacts with each cluster independently,addressing any requests directly to the proxies 254 a-n. In a secondembodiment of the multi-cluster file storage system 240, there is anadditional multi-cluster proxy 262 or multi-cluster ring 264. Themulti-cluster proxy 262 is used to provide a single entry point to theclusters 250 a-n, and the multi-cluster ring 264 is used to balancerequests across the clusters 250 a-n. An embodiment may use either amulti-cluster proxy 262, or a multi-cluster ring 264, or both, orneither. In an embodiment in which a multi-cluster proxy 262 or amulti-cluster ring 264 is used, the individual clusters 250 a-n canoptionally forego the use of the cluster-specific proxy 254 a-n orcluster-specific ring 256 a-n.

Referring now to FIG. 5, the user 202, which is exemplary of a pluralityof users that use the file storage system 100, has a user account 300with the file storage system 100 to store and receive data objects, andthat user 202 may create a plurality of containers 302 in the useraccount 300 and store a plurality of data objects 304 in each of thecontainers 302 for retrieval. In the discussion below, a user account isreferred to as an “account,” a container is referred to as a“container,” and a data object is referred to as an “object” for clarityof discussion. One of skill in the art will recognize that the terms“account,” “container” and “object” are generic forms of data namingthat are used to direct the file storage system 100 to a specific dataobject. When other types of rings and services are used, an appropriatename may be substituted. For clarity, discussion of alternative ringsand services will be limited to the “account,” “container” and “object”rings and services. The account as shown in FIG. 5, as well as otherrings and services, are consistent whether or not they are deployed on alogical structure within a single cluster, such as the structureillustrated in FIG. 3, or are arrayed across a multi-cluster system,such as the structure illustrated in FIG. 4. When reference is made tothe proxy 204, the rings 206, the services 208, 210, or 212, or thestorage pools 214, equivalent structures are also contemplated withineach cluster 250 a-n (specifically the proxies 254 a-n, the rings 256a-n, the object/container services 258 a-n, and the storage pools 260a-n). Similarly, equivalent structures to the proxy 204 and the rings206 are contemplated relative to the multi-cluster proxy 262 and themulti-cluster ring 264.

The components of the exemplary file storage system 100 and some oftheir functions will now be described in detail relative to variousembodiments.

The Rings 206

As discussed above, the rings 206 are implemented in a tailoredelectrical circuit or as software instructions to be used in conjunctionwith a processor to create a hardware-software combination thatimplements the specific functionality described herein. To the extentthat software is used to implement the rings, it may include softwarethat is stored on a computer-readable medium location in the storagemanagement server 106 and/or the storage servers 108. Referring back toFIG. 2, the rings 206 include semiconductor circuits and/orcomputer-executable instructions that, when executed by a processor,provide subsystems of the file storage system 100 that provide a mappingbetween the entities stored in the file storage system 100 and thelocations of those entities in the storage pools 214. In the illustratedembodiment, the file storage system 100 includes a separate object ring206 a, container ring 206 b, and account ring 206 c, and when componentsof the file storage system 100 need to perform any operation on anobject, container, or account, those components interact with the objectring 206 a, container ring 206 b, and account ring 206 c, respectively,to determine the location of that stored entity in the storage pools214. However, one of skill in the art will recognize that different ringstructures may be provided (e.g., a single ring for the objects,containers, and accounts, more than one ring for each of the objects,containers, and account, etc.) without departing from the scope of thepresent disclosure. The rings 206 maintain the availability and safetyof data in the file storage system 100 through the use of zones,partitions, replicas, and the storage pools 214, as described below.

A zone is defined as one or more of the storage pools 214 that aresubject to a correlated loss of access or data as a result of aparticular event. For example, each storage server 108 in the filestorage system 100 may be defined as a storage pool in a separate zone,as each storage server 108 is subject to loss of access to its storedobjects as a result of a storage device failure, a catastrophic event atthe location where the storage server resides, and/or a variety of otherobject access-loss scenarios known in the art. For the same reasons, adrive in a storage server 108 may be defined as a storage pool in aseparate zone, a plurality of storage servers 108 in a given storagerack or cabinet as a storage pool in a separate zone, a plurality ofstorage servers 108 coupled to the same switch as a storage pool in aseparate zone, a plurality of storage servers 108 in a given datacenteras a storage pool in a separate zone, a plurality of storage servers 108connected to a common power system as a storage pool in a separate zone,etc. One of skill in the art will recognize that the examples of zonesprovided above are not limiting, and a variety of zones known in the artwill fall into the scope of the present disclosure.

Logically, a partition is an abstract storage bucket. As discussed infurther detail below, the file storage system 100 maps each partition toa plurality of storage pools 214 that are in different zones, and storesdata using those partitions. The mapping of a given partition to aplurality of storage pools 214 creates a plurality of partition replicasof that partition (e.g., equal to the number of storage pools 214 thepartition is mapped to.) For example, when a given partition is mappedto 3 storage pools 214 that are in different zones, 3 partition replicasof that partition are created.

The object ring 206 a for the management of objects will be described indetail below. However, one of skill in the art will recognize how thediscussion may be applied to the container ring 206 b, the account ring206 c, and/or a ring for any other stored entity, without departing fromthe scope of the present disclosure.

In various replicated, network-based file storage systems, an objectfrom a user is received by a proxy. To determine where the object shouldbe stored, some attribute of the object or the object data itself ishashed. If necessary, some attribute of the object is modified so thatthree different results are returned from the hashing function. Theobject is then replicated and stored in the storage pool correspondingto the number returned by the hash function.

Under typical circumstances, a consistent hashing function is used asthe hash function. The use of the consistent hashing function ensuresthat there will be minimal changes to the assigned storage pools given achange in membership due to adding or removing new storage pools.

Although the consistent hashing function results in minimal changes tothe storage location, sometimes the assignments made by the consistenthashing function or the rearrangements needed due to a change inmembership may have undesirable storage characteristics. For example,such methods have been found to result in multiple object replicas forthe same object being stored in one or more storage pools that are inthe same zone. As discussed above, this is undesirable because thenmultiple (and possibly all) object replicas for the same object aresubject to being lost as a result of a particular event. Alternatively,rebalancing the replicas due to a change in membership has been found torequire the movement to two of the replicas 4% of the time, and themovement of all three replicas 1% of the time. It is desirable to neverhave to move more than one replica at a time.

In one embodiment, the file storage system 100 solves the problem ofmultiple object replicas for the same object being stored in storagepools that are in the same zone through the use of the rings 206.Referring now to FIG. 6, a method 400 for storing stored entities isillustrated. At block 402, an object us received by a user. In oneembodiment, an object is received from the user 202 by the proxy 204.The method 400 then proceeds to block 404 where a partitionidentification is generated. In one embodiment, a consistent hashfunction is applied to the object received in block 402, and the hashfunction returns a partition identification that corresponds to apartition. The method 400 then proceeds to block 406 where the partitionassociated with the partition identification is mapped to storage poolsthat are in different zones. This mapping function is constrained sothat the physical location of the storage pools is required to have oneor more desirable properties, such as having each partition replica in aseparate zone.

There are various embodiments of the constrained mapping function. Inone embodiment, the constrained mapping function is the output of aconstraint satisfaction solver, in which the desired storagecharacteristics (such as the requirement that each replica of apartition be in a different availability zone) are inputs to the solvingfunction. The solver then uses one or more search methodologies withinthe solution space to find a storage layout that maps partitions tostorage pools in a desirable manner.

In a second embodiment, a constrained mapping function is applied toportions of the partition identification (e.g., the portions of thepartition identification that the constrained mapping function isapplied to) may be bits of the output of the original hashing functionis applied to the object. For example, the number of bits to which theconstrained mapping function is applied may be known as the partitionpower, and 2 to the partition power may indicate the partition count.The constrained mapping function is designed to return a storage poollocation for each portion of the partition identification to which it isapplied, and the storage pool locations returned for a given partitionidentification will each correspond to storage pools 214 in differentzones. These storage pool locations are then associated with thepartition identification. Thus, the partition corresponding to thepartition identification is replicated multiple times in the filestorage system 100 (i.e., a partition replica is included in eachstorage pool corresponding to the storage pool locations determined fromthe constrained mapping function.) The method 400 then proceeds to block408 where the object is stored according to the partition. The objectreceived by the user 202 in block 402 of the method 400 may then bestored according to the partition corresponding to the partitionidentification, which results in multiple object replicas for the objectbeing stored in storage pools that are in different zones in the filestorage system 100. In another embodiment, the constrained mappingfunction is used to determined storage pool locations that are indifferent zones for each partition prior to the object being received bythe user 202, discussed in further detail below.

The output of the constrained mapping function signifies a particularstorage pool where a replica of the partition should be stored. Anexample of this output is as follows: When an object is received fromthe user 202 at block 402 of the method 400, and at block 404 of themethod 400, a hash function is applied to the object. In one exemplaryembodiment, the user 202 provides data including anaccount/container/object name to the proxy 204, and a hash function isapplied to the account/container/object name as follows:

Hash function (account/container/object name)==123456789

Where 123456789 is the partition identification that is returned by thehash function. At block 406 of the method 400, the partition mappingnumber may then be divided into 3 parts (e.g., the first three digits,the second three digits, and the third three digits of the partitionidentification) and the constrained mapping function is applied to eachof those parts:

Constrained mapping function (123)==storage pool location (zone 1)Constrained mapping function (456)==storage pool location (zone 7)Constrained mapping function (789)==storage pool location (zone 3)

As discussed above, the constrained mapping function is designed toreturn the storage pool location (zone 1), storage pool location (zone7), and storage pool location (zone 3) that correspond to storage poolsthat are in different zones (e.g., zones 1, 3, and 7). The storage poolslocations are then associated with the partition identification:

Partition identification: (storage pool location (zone 1), storage poollocation (zone 7), storage pool location (zone 3)

Thus, the partition corresponding to the partition identification isreplicated across storage pools that are in different zones (here, zones1, 3, and 7.) At block 408 of the method 400, the object received fromthe user 202 is then stored, using the partition corresponding to thepartition identification, in each of the storage pools corresponding tothe storage pool locations returned by the application of theconstrained mapping function to portions of the partitionidentification. Thus, 3 replicas of the object received from the user202 are stored in the file storage system 100 in storage pools that arelocated in different zones (zones 1, 3, and 7.) In one embodiment, eachof the storage pool locations are IP addresses, i.e., when each of thestorage pools are separate storage servers. In one embodiment, theconstrained mapping function is a hash function. However, one of skillin the art will recognize that a variety of functions may be used toensure that each partition is mapped to storage pools that are indifferent zones without departing from the scope of the presentdisclosure.

In another embodiment, the constrained mapping function is applied tothe file storage system 100 before the object is received by the user202 at block 402 in order to accomplish the mapping of the partitions tostorage pools described above with reference to block 406 of the method400. For example, the total number of partitions and the total number ofstorage servers/storage pools in the file storage system 100 may (andtypically will) be known. With that knowledge, the constrained mappingfunction is used to map each partition in the file storage system 100 toa plurality of storage pools that are in different zones, and thatinformation is stored in a constrained mapping database. For example, aconstrained mapping database may include partitions mapped to storagepools such as:

Partition 1: storage pool location (zone 1), storage pool location (zone2), storage pool location (zone 3)Partition 2: storage pool location (zone 4), storage pool location (zone5), storage pool location (zone 6)Partition 3: storage pool location (zone 7), storage pool location (zone8), storage pool location (zone 9)

In one embodiment, the output of the constrained mapping function can besaved for optimized lookup. For example, the saved output may beembodied in a file provided to each of the storage pools 214, or storedin a database that is available for the appropriate systems to query. Ifthe saved output is contained within a file, the storage pools 214 maythen periodically check the modification time of this file and reloadtheir in-memory copies of the ring structure as needed.

Thus, when an object is received from a user 202 at block 402, the hashfunction is applied to that object to get the partition identification(e.g., partition 1, 2, or 3 in the example above) at block 404, and thenat block 406, the partition identification may then be used with theconstrained mapping database to determine the corresponding partitionand its associated storage pool locations. This embodiment allows theprocessing necessary to map partitions to multiple storage pools indifferent zones to be conducted before objects are received from usersso that such processing does not have to be conducted each time anobject is received from a user.

For example, referring now to FIG. 7, a method for building a ring 206is illustrated. At block 502, an ideal number of partitions for eachstorage pool in the file storage system is determined. In oneembodiment, the number of partitions that should ideally be assigned toeach storage pool 214 is calculated based the weight (e.g., storagecapacity) of each storage pool 214. For example, if the partition poweris 20, the ring 206 will have 1,048,576 (2²⁰) partitions. If there are1,000 storage pools 214 of equal weight, each storage pool 214 willideally be assigned 1,048.576 partitions. This may be referred to as anideal partition count, and in the example, each storage pool 214 startsoff empty with a 1,048.576 ideal partition count. The method 500 thenproceeds to block 504 where the storage pools are placed in a sortingorder. In one embodiment, the storage pools 214 are placed in a sortingorder based on their ideal partition count (e.g., highest to lowest),and this sorting order is maintained throughout the method 500 aspartitions are assigned storage pools 214. The method 500 then proceedsto block 506 where partitions are assigned to storage pools based ontheir sorting order but with a zone restriction. In one embodiment, thepartitions are assigned to the storage pool 214 with the highest idealpartition count, but subject to the restriction that the storage pool214 to which a partition is being assigned is not in the same zone asany other storage pool 214 that includes a partition replica for thatsame partition. The method 500 then proceeds to block 508 where thesorting order of the storage pools is adjusted. In one embodiment, oncea partition is assigned to a storage pool 214, that storage pool 214will have its ideal partition count decremented and thus that storagepool 214 is moved to a lower position in the sorting order, and themethod 500 then returns to block 506 to continue to assign partitions tostorage pools based on their sorting order but with the zonerestriction. In such a manner, each partition is assigned multiplestorage pools 214 in different zones, and thus objects received fromusers may have multiple object replicas stored in storage pools indifferent zones simply by associating those objects with the partitions.

As mentioned above, another problem relates to the rebalancing of objectreplicas stored in the file storage system due to changing membership(i.e., adding or subtracting storage servers or storage pools from thefile storage system.) Such methods have been found to require the movingof multiple object replicas of the same object in response to amembership change, which is undesirable.

In one embodiment, the mapping of partitions to multiple storage poolsin different zones in the file storage system 100 described above solvesthese problems. The use of the constrained mapping function to ensurethat each partition is mapped to storage pools in different zonesensures that object replicas for the same object are never located instorage pools 214 that are in the same zone (i.e., because any givenobject received from a user is stored in a partition that is replicatedin storage pools that are in different zones.) For example, with eachstorage server 108 defined as a separate zone, the addition orsubtraction of a given storage server 108 from the file storage system100 thus can only effect one partition replica, and hence one objectreplica of a given object (i.e., because only one of the partitionreplica will ever be located on a storage server that is defined as aseparate zone.) In similar fashion, the rebalancing associated withchanging the zone membership can be accomplished without affecting morethan one replica because each zone is guaranteed to only contain onereplica of a given partition.

Periodically, partitions may need to be reassigned to different storagepools 214, and the reassignment of partitions will result in thebuilding of a new ring from an old ring. Such an event may occur due tothe removal and/or addition of a storage pool 214 from the file storagesystem 100 (e.g., a membership change.) Referring now to FIG. 8, amethod 600 for reassigning partitions in response to the removal of astorage pool is illustrated. The method 600 begins at block 602 wherethe ideal number of partitions for each storage pool is recalculated. Inone embodiment, the ideal partition count for the storage pools 214remaining in the file storage system 100 (subsequent to the removal of astorage pool) is recalculated. The method 600 then proceeds to block 604where the storage pools are placed in a sorting order as described abovewith reference to block 504 of the method 500. The method then proceedsto block 606 where partitions to be reassigned are grouped. In oneembodiment, a partition list for the partitions to be reassigned iscreated. For example, any storage pools 214 that have been removed fromthe filesystem 100 may have all their assigned partitions unassigned andadded to the partition list, and any storage pools 214 that have morepartitions than their ideal partition count may have random partitionsunassigned from them and added to the partition list (i.e., such thatthose storage pools have a number of partitions that is within apredetermined amount of their ideal partition count.) The partitions onthe partition list may then be reassigned to the storage pool 214 inblocks 608 and 610 of the method 600 substantially as discussed abovewith reference to blocks 506 and 508 of the method 500. In oneembodiment, at block 608 of the method 600, whenever a partition isreassigned to a storage pool 214, the time of the reassignment isrecorded. Reassignment times may be used when gathering partitions toreassign to storage pools 214, such that no partition replica for agiven partition is moved twice in a predetermined amount of time.However, such reassignment restrictions based on reassignment times maybe ignored for partition replicas on storage pools 214 that have beenremoved from the file storage system 100, as removing a storage pool 214only happens upon storage pool 214/storage server 108 failure and thusrequires the reassignment of the partitions.

In one embodiment, the method 600 is conducted periodically to helpbalance the amount of data stored by storage pools 214 in the filestorage system 100. For example, the partition reassignment method 600discussed above may repeated until each storage pool 214 is within apredetermined threshold of a predetermined storage capacity (e.g.,within 1% of 60% storage capacity for that storage pool) or when it isdetermined that partition reassignment will not improve the balance ofdata stored by the file storage system 100 by more than a predeterminedamount. For example, if a first storage server 108 includes 2 TB ofstorage, a second storage server 108 includes 4 TB of storage, and athird storage server 108 includes 6 TB of storage, data balancing may beconducted to ensure that each of the storage servers 108 holds the samepercentage of its storage capacity (i.e., the first storage server 108holds 0.66 TB of data, the second storage server 108 holds 1.33 TB ofdata, and the third storage server 108 holds 2 TB of data such that eachof the storage servers 108 is at 33% of its storage capacity.) Weightsmay be applied to storage servers 108 to balance the distribution ofdata on the storage servers 108 in the file storage system 100 toaccount for different storage capacities.

Object Service 208

As discussed above, the object service 208 is implemented in a tailoredelectrical circuit or as software instructions to be used in conjunctionwith a processor to create a hardware-software combination thatimplements the specific functionality described herein. To the extentthat one embodiment includes computer-executable instructions, thoseinstructions may include software that is stored on a computer-readablemedium located in the storage management server 106 and/or the storageservers 108. The object service 208 may include instructions that, whenexecuted by a processor, provide object storage and objectionmanipulation functionality such that the object service 208 is operableto, for example, store, retrieve and delete stored objects in thestorage pools 214. In one embodiment, an object service 208 is providedfor each storage pool that holds object data. For example, an objectservice 208 may be included on a server that further includes one ormore storage drives that provide a storage pool for objects. In oneembodiment, the objects are stored as binary files with metadata storedas extended attributes of the file in the filesystem used by the objectstorage service. In such an embodiment, the object service 208 will usesthe extended attributes of the filesystem to manage the metadata. In asecond embodiment, the metadata is stored in a machine-readable formatnext to the data itself. For example, the metadata for a file is storedin a text file or single file database.

In one embodiment, objects are stored by the object service 208 using apath derived by applying a hash function to the name of the object alongwith a timestamp. For example, an incoming object for a user account tobe written to a container will have a hash applied to itsaccount/container/object name and the path generated for the object is:

/objects/<partition>/<storage pool location>/objectname_hash.15673.datawhere “objects” indicate that the object data is stored in an objectstorage pool 214, <partition> is the partition identification that mapsthe object to a partition, <storage pool location> is the storage poollocation that maps the partition replica to a storage pool 214 in adifferent zone than its related partition replicas, objectname_hash isthe hash of the account/container/object name, and 15672 is thetimestamp.

When there is a request for an object, the file storage system 100 willfind all the object replicas in the file storage system 100 that includethe objectname_hash and return the object data that has the most recenttimestamp value. Special care is needed to record updates that should bepersisted as the new canonical value. For example, when an objectreplica is deleted, a modification sentinel (e.g., a 0 byte “tombstone”file or “.ts” file) is written to the storage pool 214 where the deletedobject replica was located and that includes the same objectname_hash asthe deleted object replica (i.e., /objectname_hash.15784.ts) and thattombstone file stays in the file storage system 100 for a predeterminedamount of time (e.g., 7 days.) During object replication, discussed infurther detail below, when the file storage system 100 encounters atombstone file, the file storage system 100 checks whether the tombstonefile has been in the system for 7 days. If not, the file storage system100 searches for and deletes any object replicas that it finds relatedto that tombstone file (e.g., replicas that same objectname_hash as thetombstone file) to ensure that objects that were meant to be deletedfrom the file storage system 100 are removed and older versions ofobject replicas of a given object do not appear in the file storagesystem 100 due to, for example, the temporary failure of a storageserver 108 or storage pool 214 that might have prevented the deletion ofthat object replica previously. If the file storage system 100determines that a tombstone file has been in the file storage system 100for longer than the predetermined time, that tombstone file is deleted.

The mechanism used for recording file deletion is also used to recordother types of updates. For example, a “purge” marker indicates that thesystem should overwrite all copies of the object and set the space tofree; a “version” marker indicates that the system should create a copyand mark the copy with a version number; and a “ttl” (time-to-live)marker indicates that the system should check an authoritative sourcefor updates after the expiry of a set time period. Other types ofout-of-band changes to the file are also contemplated.

Container Service 210

As discussed above, the container service 210 is implemented in atailored electrical circuit or as software instructions to be used inconjunction with a processor to create a hardware-software combinationthat implements the specific functionality described herein. To theextent that one embodiment includes computer-executable instructions,those instructions may include software that is stored on acomputer-readable medium located in the storage management server 106and/or the storage servers 108. The container service 210 may includeinstructions that, when executed by a processor, provide containerstorage and container manipulation functionality such that the containerservice 210 is operable to store, retrieve and delete stored containersin the storage pools 214. In one embodiment, a container service 210 isprovided for each storage pool that holds container data. For example, acontainer service 210 may be included on a server that further includesone or more storage drives that provide a storage pool for containers,and the container service 210 may include the names of containers andobjects in those containers. Thus, in one embodiment, the containerservice 210 handles the listing of containers, and does not hold thelocation where the objects are stored (e.g., the storage pool where agiven object replica resides), but rather the locations of containersthat hold the objects. The listings for the container locations may bestored as database files, and those listings may be replicated acrossthe storage pools 214 in a manner that is similar to the replication ofobjects (e.g., through their association with partitions) as discussedabove. Container storage statistics for the container service(s) 210 maybe tracked by the file storage system 100 and may include total numberof objects stored by one or more containers, the total storage providedby any given container, and/or a variety of other statistics known inthe art.

Account Service 212

As discussed above, the account service 212 is implemented in a tailoredelectrical circuit or as software instructions to be used in conjunctionwith a processor to create a hardware-software combination thatimplements the specific functionality described herein. To the extentthat one embodiment includes computer-executable instructions, thoseinstructions may include software that is stored on a computer-readablemedium located in the storage management server 106 and/or the storageservers 108. The account service 212 may include instructions that, whenexecuted by a processor, provide account storage and accountmanipulation functionality such that the account service 212 is operableto store, retrieve and delete stored accounts in the storage pools 214.In one embodiment, an account service 212 is provided for each storagepool that holds account data. For example, an account service 212 may beimplemented by a server that includes storage drives that provide astorage pool for accounts, and the account service 212 may include thenames of accounts and containers in those accounts. Thus, the accountservice 212 is very similar to the container service 210, discussedabove, with the exception that account storage 212 handles the listingsof accounts.

Other Services

As discussed above, other types of services may be implemented insimilar fashion to the object, container, and account services describedabove. For example, one implementation includes an authorizationservice. The authorization service may include instructions that, whenexecuted by a processor, handle the storage and manipulation ofauthorization metadata so that the authorization service is operable tostore, retrieve, delete, and query stored credentials from in thestorage pools 214. In one embodiment, an authorization service providesan ACL-based authorization. In a second embodiment, the authorizationservice provides posix-compatible authorization. In a third embodiment,the authorization service provides tree or graph-based authorization,such as would be provided with an LDAP-based authorization service.

A second implementation includes a structured data service. Thestructured data service may include instructions that, when executed bya processor, provide the storage and manipulation of structured datasuch that the structured data service is operable to store, retrieve,delete, and query tabular, graph, or tree-based data from in the storagepools 214. In one embodiment, a structured data service provides aJSON-based output. In a second embodiment, the structured data serviceprovides XML-based output. In a third embodiment, the structured dataservice provides HTML output.

Proxy 204

The proxy 204 is implemented in a tailored electrical circuit or assoftware instructions to be used in conjunction with a processor tocreate a hardware-software combination that implements the specificfunctionality described herein. The proxy 204 is responsible for tyingtogether the file storage system 100. For each request received from auser, the proxy 204 determines the location of the account, container,or object in the appropriate ring 206 (e.g., the object ring 206 a, thecontainer ring 206 b, or the account ring 206 c) and routes the requestaccordingly. A public Application Programming Interface (API) may beexposed to users through the proxy 204. A large number of failures maybe handled by the proxy 204. For example, if a storage server 108 and/orstorage pool 214 is unavailable for an object PUT, the proxy 204 may usethe rings 206 to determine an appropriate storage server 108 and/orstorage pool 214 for that object and route the object there instead. Inone embodiment, when objects are streamed to or from a storage server108, they are streamed directly through the proxy 204 and proxy server106 to or from the user 202 and/or user device 102, and are not spooledby the proxy 204 and the proxy server 106.

In another embodiment, there are multiple proxies associated with a filestorage service. The existence of multiple proxies may be ascertainablefrom outside the file storage service, or it may be transparent.Different proxies may be used for different purposes. For example, inone embodiment different proxies are used for different types of files.In another embodiment, different proxies are used for different types ofrequests. In a third embodiment, an appropriate proxy is chosen tominimize latency, geographic, or network distance between the proxy andthe system making the request.

In the context of a multi-cluster system such as the system described inFIG. 4, the multiple proxies may be used to provide a single point ofentry for the multi-cluster system, in addition to one or more proxiesat the per-cluster level.

In one embodiment, one of the functions performed by the proxy istime-stamping or logging all requests into the storage system. Thetimestamps on the incoming requests are stored as metadata and are usedin part to determine the most recent version of a file.

In an embodiment with more than one proxy, it is possible for more thanone request to come in within a short period of time. In that case, itis important to resolve any conflicts associated with multiplesimultaneous accesses to the file storage system. In one embodiment, analgorithmic approach for ordering the actions of different independentactors is used, such as the use of a vector clock. In a secondembodiment, an independent arbiter is used to resolve conflicts, usingan implementation of the Paxos algorithm or the Byzantine Generalsalgorithm.

A third embodiment may also be used to simplify and increase the speedof the system by reducing the applicable error window and thenalgorithmically picking a “winner” in the case of multiple conflictingaccesses within the error window. For example, an embodiment may use atime synchronization server and time synchronization code to reduce theclock skew between different computers in a pool, in a zone, or acrosszones to within a specified ε, for example one millisecond or onemicrosecond. The applicable ε can be determined by analyzing the patternof accesses over time, and it may be different for different accounts,different types of services, or at different times. In this case, thealgorithmic complexity of absolutely ordering the actions across thecluster is traded for the operational complexity of handling timesynchronization across the system.

Given the clock synchronization across the pools or zones within ε, thepreviously-discussed timestamp will be sufficient to absolutely orderthe actions of other systems interacting with the file storage system ifthere are not multiple conflicting accesses to the same stored valuewithin ε. In that case, the timestamping is used to order the actionsand pick the most recent version of any information or data.

If there are multiple conflicting accesses within ε, then the systemincludes a synchronization rectifier that algorithmically breaks the tieand chooses a winner. In one embodiment, this is handled by assertingthat the first copy to replicate (as further discussed below) wins. In asecond embodiment, the inconsistency is handled by throwing an error formanual intervention. In a third embodiment, the inconsistency is handledby examining one or more additional types of metadata, such as latencyto the originating user (given identical arrival times, the originatingserver with lower latency issued second), internal file modification orcreation times, or an arbitrary ordering on an unrelated value. A fourthembodiment uses geolocation on the requesting IP address and allowingthe request that is geographically closer to the timestamping gateway. Afifth embodiment detects the attempted simultaneous access and does notallow any write, instead returning an error in response to bothrequests. A sixth embodiment evaluates the requests to see if they canbe merged and both accesses granted.

In the context of a multi-cluster system such as the system described inFIG. 4, the latency between clusters is much more likely to be higher.This can be dealt with by increasing ε or by only synchronizing accesseswithin a single cluster.

Replicators

Replicators are implemented in a tailored electrical circuit or assoftware instructions to be used in conjunction with a processor tocreate a hardware-software combination that implements the specificfunctionality described herein. To the extent that one embodimentincludes computer-executable instructions, those instructions may beimplemented as software stored on a computer-readable medium located inthe storage management server 106 and/or the storage servers 108, andmay include instructions that, when executed by a processor, keep thefile storage system 100 in a consistent state in the face of temporaryerror conditions like network outages, storage pool 214 failure, and/orstorage server 108 failure. For example, an object replicator may beprovided for each storage pool 214 (e.g., a storage server 108 thatprovides a storage pool) that holds object data. The replicators comparestored entities in their storage pool 214 with each replica of thatstored entity in other storage pools 214 in the file storage system 100to ensure that all related replicas contain the latest version of thestored entity. In one embodiment, object replicators may use a hash listto quickly compare subsections of partitions, while containerreplicators and account replicators may use a combination of hashes andshared storage account metadata. In one embodiment, replicator updatesof stored entities are push based. For example, replicators may comparethe replica stored entities in their storage pools 214 with relatedreplica stored entities in other storage pools in the file storagesystem 100, and if the replicator determines there is a differencebetween the replicas (e.g., by applying an order independent check sumto the related replicas), the replicator may then push the data thatrelated replica stored entities in other storage pools need in order tobe up to date. In one embodiment, the pushed updates include rsyncingreplicas to efficiently provide only the data needed by the out-of-datereplica. Account and container replicators may either push missing dataover HTTP or rsync whole database files in the event it is determinedthat a push update will be inefficient. The push-based updates discussedabove results in replicas being updated generally only from “local”storage pools 214 to “remote” storage pools 214. In one embodiment, thisprovides a benefit as data in a storage pool 214 may not belong there(as in the case of handoffs and ring changes), and a replicator can'tknow what data exists elsewhere in the file storage system 100 that itshould pull in. Thus, it's the duty of any replicator associated with agiven a storage pool 214 that contains data to ensure that data gets toother storage pools where it belongs. As discussed above, replicatorsmay also ensure that data is removed from the system by creating thetombstone files as the latest version of a replica when that replica isdeleted, and then search out and removing all replicas related to thattombstone file from the file storage system 100.

Database Replicators

Database replicators are a type of replicator, discussed above, thatoperate on storage pools 214 that contain accounts or containers (i.e.,there may be account replicators and container replicators.) To performthe replication discussed above, the first step that a databasereplicator may perform may be a low-cost hash comparison to find outwhether or not two replicas (e.g., a replica on the database replicatorslocal storage pool 214 and a related replica on a “remote” storage pool214) already match. Under normal operation, the hash comparison allowsrelatively quick verification that databases in the file storage system100 are already synchronized. If the hashes differ, the databasereplicator may bring the databases in sync by sharing records addedsince the most recent previous sync point. This most recent previoussync point notes the last record at which two databases were known to bein sync. After all new records have been pushed to the remote database,the sync table (which lists which remote databases a local database isin sync with) of the local database is pushed to the remote database, sothe remote database knows it's now in sync with database that the localdatabase has previously synchronized with. If a database replica (e.g.,an account replica or container replica) is found to be missing entirelyfrom a storage pool 214 that it should exist in, the entire localdatabase file may be recreated on that storage pool 214 using rsynctechniques known in the art. In one embodiment, when an entire localdatabase file is be recreated on a storage pool 214 using rsync, thatdatabase may be vested with a new unique id.

Object Replicator

Object replicators are a type of replicator, discussed above, thatoperate on storage pools 214 that contain objects. In one embodiment,object replicators associated with a storage pool 214 may performedrsync techniques known in the art on remote storage pools to determineappropriate data to push data to remote storage pools. However, asobject replication times may increase using this method when the filestorage system 100 gets sufficiently large, a hash of the contents foreach suffix directory may instead be saved to a per-partition hashesfile, and the hash for a given suffix directory is then invalidated whenthe contents of that suffix directory are modified. The objectreplicator may then read these hash files, calculate any invalidatedhashes, and transmit the hashes to each remote storage pool 214 thatshould hold the partition, and only suffix directories with differinghashes on the remote server are then rsynced. After pushing data to theremote storage pools 214, each rsynced suffix directory has its hashesrecalculated. Object replicator performance is generally bound by thenumber of uncached directories it has to traverse, usually as a resultof invalidated suffix directory hashes. In one embodiment, the filestorage system 100 is designed so that around 2% of the hash space on anormal storage pool 214 will be invalidated per day.

Updaters

Updaters are implemented in a tailored electrical circuit or as softwareinstructions to be used in conjunction with a processor to create ahardware-software combination that implements the specific functionalitydescribed herein. To the extent that one embodiment includescomputer-executable instructions, those instructions may includesoftware that is stored on a computer-readable medium located in thestorage management server 106 and/or the storage servers 108, and mayinclude instructions that, when executed by a processor, process updatesthat may have failed. An updater may be provided with each storage pool(e.g., on a server that includes the storage pool) to process failedupdates. For example, there may be times when container or account datawill not be immediately updated. Such incidents may occur during failurescenarios or periods of high load. If an update of a stored entityfails, the update is queued in a storage pool 214 on the file storagesystem 100, and the updater that is associated with that storage pool214 will process the failed updates. In such situations, a consistencywindow is used. For example, suppose the container service 210 is underload and a new object is put in to the file storage system 100. Theobject will be immediately available for reads as soon as the proxy 204responds to the user 202 that the object has been successfully added tothe file storage system 100. However, due to the heavy load, a containerservice 210 may not have been able to update its object listing, and sothat update would be queued for a later update. Container listings,therefore, may not immediately contain the object, although the objecthas been saved and replicated within the applicable object storage poolarea. In one embodiment, the consistency window needs only to be aslarge as the frequency at which the updater runs.

Multi-Cluster Container Synchronization

In the context of a multi-cluster system such as the system described inFIG. 4, multi-cluster replication and synchronization is necessary. Themulti-cluster synchronization systems build on the single-clustersystems described above, but adapt for the differences in latency andcontrol that are likely between different clusters. To the extentapplicable, the multi-cluster systems include similar services,components, and capabilities to those described above.

Multi-cluster replicators are implemented in a tailored electricalcircuit or as software instructions to be used in conjunction with aprocessor to create a hardware-software combination that implements thespecific functionality described herein. To the extent that oneembodiment includes computer-executable instructions, those instructionsmay be implemented as software that is stored on a computer-readablemedium located in the storage management server 106 and/or the storageservers 108 within each cluster 250 a-n, and may include instructionsthat, when executed by a processor, keep the elements of themulti-cluster file storage system 240 in a consistent state acrosscluster boundaries.

The multi-cluster replicators can be invoked abstractly, in a mannerconsistent with the intra-cluster replicators, but with reducedimmediate availability guarantees due to increased latency and highersynchronization overhead. This has the advantage of being transparent tousers of the file storage system, but in some instances may also be lesstransparent in its operation and limitations.

For example, one embodiment of the multi-cluster file storage system 240allows for a configurable number of remote replicas. In this embodiment,the number of such remote replicas may be specified to the multi-clusterreplication system using appropriate attributes of the request(s)associated with establishing the remote replication target(s). To theextent that specifying the remote replica count on a per container basisadds complexity to the replication system, the incorporatedfunctionality allowing cluster-wide replica counts can be leveraged tocount remote replicas, with but with less particularized control overthe location of and circumstances of each replica. Thus, in analternative embodiment, the remote replica count is not per seconfigurable, and the number of replicas the remote cluster isindirectly configured as part of the total replica count.

Note that multi-cluster replication and synchronization will in mostcases be a per-account feature, with granularity down to a container orobject level. While certain guarantees can be made about replication bythe cloud services provider, most times customers will not want to payfor widely distributed storage—either in cost, or in the increasedlatency and decreased performance associated with widely dispersedreplicas. This is especially so when the different clusters are providedby more than one organization, such as live mirroring of important databetween two independent cloud service providers, perhaps in differentregions. In that case, an end user will need to have accounts andcontrol the interaction between the two different service providers. Inthese cases, increased control over specific inter-cluster replicationand synchronization characteristics is not only desirable but necessary.Accordingly, the multi-cluster replication system has alternativeembodiments that allow more discrete control over inter-clusterreplication and synchronization as described below.

FIG. 9 shows a method 700 of specifying container synchronizationtargets and authorization using X-Container-Sync-To andX-Container-Sync-Key attributes to control the container synchronizationmechanism. In the discussion below, the information associated with theaccount and authorization is included in the headers of the request, thedata in the body of the request, and the target specified via a URL.Various embodiments of this functionality will be described relative tothe “st” tool, a command that generates HTTP/1.1-compliant GET, POST,DELETE and similar requests. Similar HTTP request functionality isavailable and known in the art. Other available mechanisms includecommunications protocols known in art such as, without limitation,rsync, XMPP, SIP, SOAP, XML-RPC, or CORBA.

The method begins at step 702, including setting a first container'sX-Container-Sync-To attribute to a second container's URL, and theX-Container-Sync-Key attribute to a key value. This can be achievedusing the following command:

$ st post-t https://cluster2/v1/AUTH_[user]/container2-k [secret value]container1

Step 704 includes setting a second container's X-Container-Sync-Toattribute to first container's URL, and the X-Container-Sync-Keyattribute to the same key value as the first container. This providesthe embodiments of the object storage system the identification of thesynchronization target and corresponding secret key. This purpose can beachieved with the following command:

$ st post-t https://cluster1/v1/AUTH_[user]/container1-k [secret value]container2

After steps 702 and 704, the objects in the first and second containersare set to be synchronized but are not yet synchronized. The actualreplication and synchronization is done at step 706 using the containerreplication process. The container replication process uses thecontainer replication and object replication structures described withreference to the intra-cluster replication and can optionally beperformed by the same process or by other specialized replicatorsrunning in parallel. In this embodiment the container replicationprocess is performed on a periodic basis. If the synchronization of step706 is performed on a periodic basis, then the system cannot provide aguarantee as to when a change will make it to the remote cluster. Forexample, a successful completion of the PUT request indicates thatcluster has the object, not the remote cluster. The synchronization ofthe containers happens in the background.

In other embodiments the replication and synchronization process runbased upon the occurrence of an event, such as the specification of acontainer to synchronize to (performing initial synchronizationsimultaneously) or on the addition of a new object to a container. Thiscan provide higher-quality guarantees, but individual actions may takelonger to complete and may see a higher request failure rate.

In one embodiment, container synchronization mechanism at step 706 ismanaged by one or more daemon processes that control the synchronizationof the data objects between individual container servers. Daemonprocesses are computer processes that run in the background, rather thanunder the direct control of a user; and that are usually initiated asbackground processes.

In one embodiment, a daemon runs on each container server. The daemonscans every container database looking for containers that areconfigured to sync, and sends synchronization updates to proper entitybased on any newly detected ROWIDs in the container database thatrepresent containers that need to be synchronized. In anotherembodiment, a single daemon runs per cluster that managessynchronization for all servers in the cluster. In yet anotherembodiment, one daemon handles synchronization for a configurable numberof servers.

In one embodiment, each daemon process may be configured to only spend acertain predetermined amount of time trying to sync to a givencontainer. This avoids one frequently updated container causing updatesfor other containers to not be processed or delayed. In variousembodiments, this amount of time may be the same or different for eachcontainer to be synced. In one or more embodiments, the aforesaid amountof time may be determined based on an individual synchronizationfrequency or priority attribute value assigned to the individualcontainer to be synchronized during the sync configuration process.

In one embodiment, the daemons managing the container synchronizationmechanism are coordinated to ensure that synchronization messagesbetween container servers are always sent. For example, if a specificcontainer is synced to three different container servers, there arethree possible daemons that can cause a synchronization message to begenerated. If each daemon only does one third of the synchronizations,this one third of synchronizations will be lost if one of the containerservers executing the aforesaid daemon is down. On the other hand, ifall three daemons are configured send all the synchronization messages,then the system will be inefficient due to over-redundancy. Therefore,in one embodiment, the aforesaid three daemons are configured such thateach daemon sends two thirds of the possible synchronization messages.In this configuration, the aforesaid double redundancy assures that nosynchronization messages will be lost if one of the three containerservers becomes inoperable.

In one or more embodiments, the container synchronization is performedby the container synchronization daemons in the pass-by-pass manner.Each execution of the daemon results in one round of replication pass.In one or more embodiments, the container synchronization daemonaccumulates and logs various statistics related to the currently runningreplication pass.

In one embodiment, the daemon on each container server keeps sync pointsin the local container databases that control how synchronizationmessage are sent. For example, the daemon may keep a first sync pointfor the newest ROWID known and a second sync point for the newest ROWIDfor which all synchronization updates have been sent. In this example,any row updated that is newer than both sync points will cause asynchronization update to be sent if the corresponding container serveris designated to send that update. Typically, if three container serversare configured to synchronize a container, each server will handle onethird of the synchronizations. In addition, if any row between the twosync points is updated, all three servers will send a synchronizationmessage. This ensures that such synchronizations will be processed evenif a server is down.

An exemplary operational scenario of the inventive container replicationmechanism in accordance with one embodiment of the invention will now bedescribed. The aforesaid exemplary operational scenario assumes aconfiguration with three container replicas as well as perfectlymatching ROWIDs starting at 1. During the first containersynchronization run, the database stores six rows:

-   -   SyncPoint1 starts as −1.    -   SyncPoint2 starts as −1.    -   No rows between points, so no “all updates” rows.    -   Six rows newer than SyncPoint1, so a third of the rows are sent        by node 1, another third by node 2, remaining third by node 3.    -   SyncPoint1 is set as 6 (the newest ROWID known).    -   SyncPoint2 is left as −1 since no “all updates” rows were        synced.

In the next container synchronization run, the database has 12 rows:

-   -   SyncPoint1 starts as 6.    -   SyncPoint2 starts as −1.    -   The rows between −1 and 6 all trigger updates (most of which        should short-circuit on the remote end as having already been        done).    -   Six more rows newer than SyncPoint1, so a third of the rows are        sent by node 1, another third by node 2, remaining third by node        3.    -   SyncPoint1 is set as 12 (the newest ROWID known).    -   SyncPoint2 is set as 6 (the newest “all updates” ROWID).

In the described way, under normal circumstances, each node sends itsshare of updates each run and just sends a batch of older updates toensure nothing was missed.

As would be appreciated by those of skill in the art, a crash of acontainer server may cause lost container database copies. In anembodiment, the aforesaid lost copies are replaced by one of theremaining container database copies on the other container servers. Thereestablished server then receives the sync point information from thecontainer database copy. As would be appreciated by those of skill inthe art, no updates are lost due to the described “all updates”algorithm the other two container servers use. It should be also notedthat rebalancing the container ring moves container database copiesaround, but results in the same behavior as a crashed server would.

As would be appreciated by those of skill in the art, in bi-directionalsynchronization configurations, the receiving container server will sendback the received updates to the sending container server. To preventthis situation, one embodiment operates to track where synchronizationupdates were received from and prevent sending synchronization back totheir original sender.

Step 708 uses a container replication auditor to verify correctsynchronization and replication of the data. While this step isoptional, it can be used to ensure better data availability andreplication confidence levels when using widely-dispersed clusters thatdon't support real-time replication guarantees. The containerreplication auditor can operate at a metadata level, checking only filenames and file sizes, or it can use cryptographic or content hashes todetermine differences with very high confidence. In one embodiment, thecontainer replication auditor pre-computes the hashes for the containerusing the rsync algorithm so that cross-network synchronization usingrsync is accomplished more quickly.

FIGS. 10-12 show three different container replication structures thatcan be created. These container replication structures are onlyexemplary and not limiting. More specifically, these three basicstructures can be expanded or combined to provide any requirereplication structure.

FIG. 10 shows the most basic reciprocal replication, with just twocontainers, 802 and 804. Each of these containers is set up to replicateto the other. The replication structure of FIG. 10 can be used toperform simple replication of important data between two geographicallydispersed clusters or two clusters in different organizations. Withineach cluster, the existing replication structures would provideintra-cluster redundancy based on the ring structure.

FIG. 11 shows a second embodiment with multiple containers linked in asynchronization chain. In such a configuration, a first container 806 isconfigured to sync to a second container 808, which is then configuredto sync to a third container 810. The third container 810 is thenconfigured to sync to the first container 806, completing thesynchronization chain. In such an embodiment, an arbitrary number ofcontainers may be included in the aforesaid synchronization chain.

FIG. 12 shows a third embodiment with multiple containers linked in asynchronization tree. In such a configuration, a first container 812 isconfigured to sync simultaneously to a second container 814, a thirdcontainer 816, and a fourth container 818. Note that thissynchronization is one-way—there is no return synchronization. Asynchronization tree can be used for effective replication of sharedassets out to a number of distinct physical locations, such as for acontent delivery network.

Although the examples above have been described with reference tointer-cluster synchronization, another use for the synchronizationtechniques described in the above embodiments is a live accountmigration. In such a procedure, an account in a first cluster is set tosync to an account in a second cluster. The second cluster may or maynot be controlled by the same cloud service provider. After a period oftime, the account on the second cluster will be synced with the accounton the first cluster. At that point, the authentication response URL forthe account can be switched from the first cluster to the secondcluster, and all existing account tokens can be revoked. The account onthe first cluster is then put into read-only mode, and syncing from theaccount on the first cluster to the account on the second cluster isturned off. After a time long enough to ensure the account on the firstcluster is no longer being used, this account may be purged.

As it would be appreciated by those of skill in the art, in one or moreembodiments, a valid replication target container, which is specified inthe X-Container-Sync-To destination attribute, must be configured foreach cluster ahead of time. In an embodiment, this feature isimplemented based on cluster trust.

In the embodiments described above, cluster trust is achieved throughthe use of a user-provided shared secret that is keyed to a particularaccount. This shared secret can be considered equivalent to a passwordand can be used to form challenge-response pairs indicating knowledge ofthe secret key by the containers on either side of the replicationtransaction.

In some embodiments, it may not be desirable to use a single shared key.For example, a synchronization tree that provided content from a singleupstream provider to a number of competitive downstream providers maynot want to use a single key providing access to all of the containersin the tree, including the upstream container. In this embodiment, apublic-private key mechanism is used, either through explicit keys orthrough the use of certificates.

FIG. 13 shows a method 900 for establishing cluster trust in apublic/private key setting. The method 900 begins at step 902, whereinthe user engages a first cluster and designates it as a trust root. Thetrust root may be arbitrary, in cases where the eventual replicationwill be circular, or it may be the cluster holding the container at theroot of the synchronization tree.

At step 904, the cluster at the trust root establishes network contactwith the cluster. Because the container synchronization can occurs atthe logical container level, access to the container data structureneeds to be reachable from the trust root. The network contact can beprovided in a number of different ways. In a first embodiment, thecontainer and object services are directly addressable from any publicnetwork, and misuse of the services is prevented by using anauthenticated API for all requests. In a second embodiment, thecontainer server is not directly addressable, but must go through aproxy. This has the benefit of hiding the internal structure of thesecond cluster from the outside and allows all requests to go throughthe same path, but it also may require more computation on the part ofthe replicator at the trust root to discover and deal with differences.In a third embodiment, there is a tunnel allowing effective directaccess between the services at the trust root and the services at thesecondary cluster, without having the services directly exposed to thepublic internet. This may be accomplished by using a tunnel server withport mapping, allowing apparently direct access that in reality istunneled to the service layer at the secondary cluster. In a fourthembodiment, there is a shared line between the first and second cluster,for especially high volume and low latency replication.

In one or more embodiments, the authentication mechanism of the objectstorage system as well as the container server reference a list of hostsindicating the allowed remote clusters for container synchronization.The embodiment operates to ensure that only the containers located onthe listed clusters are allowed to synchronize with the localcontainers.

At step 906, the trust root creates or imports an existingpublic/private keypair. It then imports the public keys from thesecondary cluster to the trust root and sends the public key from thetrust root to the secondary cluster. This allows for the cryptographicverification of identity from one cluster to another.

At step 908, the trust root connects to the secondary cluster andestablishes the trust relationship from the trust root to the secondarynode. In one embodiment, this is done using st:

$ st post-t https://cluster2/v1/TRUST_[user]-priv [root_priv_key]-pub[secondary_pub_key]

This command sets the trust relationship for a certain user between thetrust root (where this is being run) and the secondary cluster(cluster2).

In the final step 910, a new trust root or secondary cluster is chosenif necessary, and the process repeats from step 904. For example, in thecase of reciprocal replication relationships as in FIGS. 10 and 11, eachcluster would be both a trust root and a secondary cluster. In atree-based relationship as in FIG. 12, a new secondary cluster would bechosen and the trust would be established with that cluster.

Another embodiment using public/private keys leverages SSH. In anembodiment that uses SSH, each cluster is provided with a public keycorresponding to a private key held either by the user or by adesignated trust root. Using the agent forwarding functionality of SSH,each individual cluster can use the same authentication credentialswithout having the necessary reverse credentials to abuse the trustrelationship. Other embodiments can use existing authenticationinfrastructure such as OAUTH, XAUTH, X.500, LDAP, or Kerberos.

One other use for the cluster trust mechanism is for cluster federation.Referring briefly to FIG. 4, a multi-cluster ring or proxy may need tohave authentication information to associate with and interoperate withthe various clusters. The same cluster trust mechanisms described abovealso allow virtual multi-cluster, multi-vendor clouds and federationbetween clusters on one provider or in one region and clusters inanother.

As it would be appreciated by those of skill in the art, in one or moreembodiments, the replication system must provide enough networkcommunication bandwidth between the clusters in the synchronization pairor synchronization chain in order to keep up with all the changes to thesynchronized containers. In one or more embodiments, the system wouldautomatically test available network communication bandwidth betweenclusters and generate a warning or other appropriate message when theavailable bandwidth is not adequate for proper synchronization. Thisoperation could be performed when the container synchronization pair orchain is established. It should be also understood that in someimplementation, there could be a burst of bandwidth use when thesynchronization feature is turned on for an existing containercontaining a significant number of objects.

As noted above, one method for minimizing the bandwidth used forcontainer replication using rsync or a similar protocol that onlytransmits differences between two files across the network connectionbetween the first and second containers. One place where rsync is lessefficient, however, is the calculation and transmission of informationfor a large number of files. Each transmission includes a certain amountof per-file and per-connection overhead. When the average size of thereplicated object is large, the overhead is negligible compared to thetotal data transfer that needs to occur. When there are only minordifferences, or there are lots of files, the latency and overheadassociated with rsync can make it less efficient.

In various embodiments, the replication process uses a compressionscheme for packing multiple objects in a container into a single virtualfile for synchronization and replication between the clusters. In afirst embodiment, this compression scheme aggregates files based uponthe size of the disk block holding the objects. For example, if the diskblock size is 4 mb, then files smaller than 4 mb could be aggregatedinto a single virtual file and sent over as a unit. In this case, thereading and writing of the objects could be accomplished at a higherspeed with less fragmentation of the underlying disk.

In a second embodiment, objects are aggregated based on a heuristicassociated with low total differences. For example, the replicator canidentify all objects in a particular container that have the same nameand size since the last replication pass and bundle all of them into asingle virtual file. While it is possible that files with the samefilename and filesize could still have changed inside the file, thisquick heuristic would allow the many files that had not changed to besynchronized with a single network trip, reducing overhead.

A third embodiment looks at the disk blocks corresponding to aparticular container and transfers all of the underlying changes as asingle virtual file. This has the advantage of reducing the overheadassociated with the network transfer to the minimum, but has twodisadvantages. First, the objects in a container may not be in oneparticular place on the disk, and so a virtual disk image may need to becreated on the fly. Second, this relies on one single transfer to beeffective; if anything happens during the transmission of the singlevirtual file, then the entire transfer needs to be redone. A similarissue arises with large files; for files of multiple GB in size,transfer of the entire file may fail, resulting in a need to re-try thetransfer.

Accordingly, a fourth embodiment uses a sliding scale to determine theoptimal point of bandwidth usage versus transfer reliability. Thisembodiment begins with a certain target size file, such as 4 MB, anddynamically increases or decreases the target size based upon networkperformance. If there is more bandwidth to spare, then the target sizeof the transferred file can be doubled one or more times up to somelimit (such as, for example, 256 MB). The actual objects in thecontainer can be manipulated to approach the target size for thetransferred file by either combining files (in the case of files smallerthan the target transfer size) or by partitioning files (in the case offiles larger than the target transfer size). In this manner, the networktransfer can occur at a speed that optimizes for performance based uponactual observed conditions. Network conditions can be observed throughobserving throughput of the process, by analyzing ping times, or byreceiving information back from the underlying operating system relativeto the TCP window being used for the transfer.

In one or more embodiments, the container synchronization processdescribed herein is adopted to provide account synchronizationfunctionality. Specifically, account synchronization is performedbetween a first account established on a first cluster and a secondaccount established on the second cluster. To this end, two accountcontainers are created on the respective clusters and configured to syncin the manner described above. The container synchronization systemdescribed above ensures synchronization of the accounts, includingreplication of the delete and post requests to the target accountcontainer.

In an embodiment of a multi-cluster system such as the system describedabove, each cluster may be in a different zone. If a multi-cluster ringis used, ring can be used to directly address zones that are indifferent clusters transparently. If a two-tier system is used, with amulti-cluster ring as well as a per-cluster ring, then the multi-clusterring maps requests to the appropriate clusters, and the per-clusterrings map to zones within the cluster. For a multi-cluster ring, it maybe acceptable to have redundancy across a lower number of clusters, suchas two, particularly if the different clusters are in different regions;clusters in different physical regions are much less likely to havecorrelated failures and so a cluster in a second region can be used as a“hot spare.”

In this case, the multi-cluster file storage system 240 is configured toprovide greater availability and durability with geographically distinctreplicas. The concept of availability zones described above issufficient for most high-availability deployments, but rare regionalfailures are known that affect multiple availability zones locatedwithin the same geographical region. Thus, multi-region replication,wherein the same object/container/account are replicated in two or moredifferent geographical zones allows service availability in the face ofregional disruptions in service. The embodiments of the inventiveconcept accomplish this purpose by providing multi-region replicationbetween geographically different or logically distinct object storagesystems. Thus, availability of the object replicas is not affected bythe aforesaid regional disruptions in service.

Using one of the methods described above, the multi-region replicationsystem is configured to target a second object storage cluster as apartition target for replication. The aforesaid targeted second objectstorage cluster may be located within a different geographic region, asmay be automatically determined by the replication system based onappropriate attributes or other relevant stored metadata. Once thereplication target has been specified in the described manner,replication between the affected partitions occurs in as describedabove.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. Accordingly, it is appropriate that the appended claims beconstrued broadly and in a manner consistent with the scope of theembodiments disclosed herein.

1. A multi-cluster synchronization system, comprising: a first clusterincluding a first cluster-internal network, the first cluster furtherincluding a first structured information repository and a first objectstorage, wherein the first structured information repository containsmetadata corresponding to stored information objects in the first objectstorage, and wherein the first structured information repository and thefirst object storage are coupled via the first cluster-internal network;a second cluster including a second cluster-internal network, the secondcluster further including a second structured information repository anda second object storage, wherein the second structured informationrepository contains metadata corresponding to stored information objectsin the second object storage, and wherein the second structuredinformation repository and the second object storage are coupled via thesecond cluster-internal network; an intercluster network coupling thefirst cluster and the second cluster; an intercluster repositorysynchronizer coupled to the first and second structured informationrepositories; the intercluster repository synchronizer furtherincluding: a target module identifying the first cluster as areplication source and the second cluster as a replication target; aconsistency evaluation module adapted to evaluate the differencesbetween the first structured information repository and the secondstructured information repository; an intercluster replication moduleadapted to selectively replicate stored information objects in the firstobject storage in the first cluster to the second object storage in thesecond cluster based on metadata in the first structured informationrepository.
 2. The system of claim 1, wherein the second cluster is alsoidentified as a replication source, and wherein the first cluster isalso identified as a replication target, and wherein stored informationobjects in the second object storage in the second cluster areselectively replicated to the first object storage in the first cluster.3. The system of claim 2, wherein the second cluster is identified as areplication source and the first cluster is identified as a replicationtarget by a second target module associated with a second interclusterrepository synchronizer.
 4. The system of claim 1, further including athird cluster including a third cluster-internal network, the thirdcluster further including a third structured information repository anda third object storage, wherein the third structured informationrepository contains metadata corresponding to stored information objectsin the third object storage, and wherein the third structuredinformation repository and the third object storage are coupled via thethird cluster-internal network; wherein the second cluster is alsoidentified as a replication source and the third cluster is identifiedas a replication target, and wherein stored information objects in thesecond object storage in the second cluster are selectively replicatedto the first object storage in the third cluster.
 5. The system of claim4, wherein the second cluster is identified as a replication source andthe third cluster is identified as a replication target by a secondtarget module associated with a second intercluster repositorysynchronizer.
 6. The system of claim 1, further including a thirdcluster including a third cluster-internal network, the third clusterfurther including a third structured information repository and a thirdobject storage, wherein the third structured information repositorycontains metadata corresponding to stored information objects in thethird object storage, and wherein the third structured informationrepository and the third object storage are coupled via the thirdcluster-internal network; wherein the third cluster is also identifiedas a replication target, and wherein stored information objects in thefirst object storage in the first cluster are selectively replicated tothe third object storage in the third cluster.
 7. An interclusterrepository synchronizer, comprising: an information processing system,including a processor, a computer-readable medium, and a network device;a structured information repository on the computer-readable medium, thestructured information repository containing a plurality of recordscorresponding to a selected group of stored information objects; asynchronization indicator, the synchronization indicator storing anaddress associated with a remote replication target; a replicator,operable to send a message using the network device to the replicationtarget responsive to changes in the structured information repository,and further operable to receive a message that a plurality of storedinformation objects have been duplicated at the remote replicationtarget, wherein the duplicated stored information objects are selectedbased on a shared metadata indicator stored in the structuredinformation repository.
 8. The intercluster repository synchronizer ofclaim 7, further comprising a local duplicator operable to receive amessage via the network device identifying changes to make in anassociated repository of stored information objects, and to send amessage using the network device describing changes in the associatedrepository.
 9. The intercluster repository synchronizer of claim 7,wherein the synchronization indicator storing a second addressassociated with a second remote replication target, and wherein thereplicator is operable to send a message using the network device to thesecond replication target responsive to changes in the structuredinformation repository, and further operable to receive a message that aplurality of stored information objects have been duplicated at thesecond remote replication target.
 10. The intercluster repositorysynchronizer of claim 7, further comprising a trust module operable tonegotiate a connection between the intercluster repository synchronizerand the remote replication target.
 11. The intercluster repositorysynchronizer of claim 7, wherein the intercluster repositorysynchronizer is at a first location, and the remote replication targetis at a second location.
 12. The intercluster repository synchronizer ofclaim 11, wherein the first location is controlled by a first party, andthe second location is controlled by a second party.
 13. Theintercluster repository synchronizer of claim 7, wherein thesynchronization indicator is set via a message prior to anysynchronization.
 14. A method of synchronizing objects, the methodcomprising: identifying a set of stored information objects with acommon metadata value at a first location; setting a synchronizationindicator associated with a replication target, the replication targetat a second location; receiving a message identifying changes to the setof stored information objects; and sending a message to the replicationtarget; wherein sending a message to the replication target results inthe duplication of the changes at the replication target.
 15. The methodof claim 14, wherein the common metadata value is one of an account, apartition, and a container.
 16. The method of claim 14, furthercomprising the step of, prior to sending a message to the replicationtarget, establishing a secure connection to the replication target. 17.The method of claim 14, wherein the first location is a remote location,and the second location is a local location.
 18. The method of claim 14,further comprising the steps of setting a second synchronizationindicator associated with a second replication target, the replicationtarget at a third location; and sending a message to the secondreplication target, wherein sending a message to the second replicationtarget results in the duplication of the changes at the secondreplication target.
 19. The method of claim 14, wherein setting thesynchronization indicator is done through HTTP.
 20. The method of claim14, wherein the first location controlled by a first party and thesecond location is controlled by a second party.